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ADAPTATION OF CLOSE-CAPTIONED TEXT 
BASED ON SURROUNDING VIDEO CONTENT 

The present invention relates generally to displaying video content 
containing close-captioned text (alternatively referred to as "close-captioning"), and more 
particularly, to apparatus and methods for adaptation of close-captioned text based on 
surrounding video content. 

Close-captioned text is used on televisions and other monitors to display 
text corresponding to the audio portion of video content being displayed. The attributes 
(e.g., color, brightness, contrast, etc.) of the close-captioned text are fixed irrespective of 
the attributes of the video content surrounding the closed-captioned text. This is 
particularly a problem where the video content surrounding the close-captioned text is the 
same color as the close-captioned text. In other situations, a weaker contrast of the 
closed-captioned text may be preferable. For instance, very bright white text in a dark 
scene may be distracting or disturbing to a viewer. Other attributes of the video content 
surrounding the closed captioned text, such as contrast, brightness, arid the presence of 
foreground objects at the location of the close-captioned text pose additional problems. 

Therefore it is an object of the present invention to provide methods and 
devices that overcome these and other disadvantages associated with the prior art. , 

Accordingly, a method for displaying close-captioned text associated with 
video is provided. The method comprising: determining a position on a portion of the 
video for display of the close-captioned text; detecting one or more attributes of the video 
surrounding the position; and adjusting one or more attributes of the close-captioned text 
based on the detected one or more attributes of the video. 

The method can further comprise displaying the close-captioned text in the 
portion of the video with the adjusted one or more attributes. 

The one or more attributes of the video surrounding the position can be 
selected from a list consisting of a brightness, a contrast, a color, and a content. 

The one or more attributes of the close-captioned text can be selected from 
a group consisting of a brightness, a contrast, a color, and a degree of transparency. 

The detecting can comprise: scanning a predetermined number of pixels 
in the video surrounding the position; and ascertaining an attribute of the pixels with a 
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look-up table; and equating the ascertained attribute of the pixels with the one or more 
attributes of the video surrounding the position. The one or more attributes of the video 
surrounding the position can be a color and the look-up table can be a color look-up table. 

The one or more attributes of the video surrounding the position can be a 
color and the adjusting can comprise choosing a different color of the close-captioned 
text. 

The one or more attributes of the video surrounding the position can be at 
least one of brightness and contrast and the adjusting can comprise adjusting at least one 
of the brightness and contrast by a predetermined factor. The predetermined factor can 
be changeable by a user. The predetermined factor can be 50%. 

The one or more attributes of the video surrounding the position can be a 
content of the video surrounding the position and the adjusting can comprise modifying a 
transparency of the close-captioned text by a predetermined factor. 

Also provided is a device for displaying close-captioned text associated 
with video. The device comprising a processor for determining a position on a portion of 
the video for display of the close-captioned text, detecting one or more attributes of the 
video surrounding the position, and adjusting one or more attributes of the close- 
captioned text based on the detected one or more attributes of the video. 

The device can further comprise a display for displaying the video, 
wherein the processor further displays the close-captioned text in the portion of the video 
with the adjusted one or more attributes. 

The one or more attributes of the video surrounding the position can be 
selected from a list consisting of a brightness, a contrast, a color, and a content. 

The one or more attributes of the close-captioned text can be selected from 
a group consisting of a brightness, a contrast, a color, and a degree of transparency. 

The device can be selected from a list consisting of a television, a monitor, 
a set-top box, a VCR, and a DVD player. 

Also provided are a computer program product for carrying out the 
methods of the present invention and a program storage device for the storage of the 
computer program product therein. 

These and other features, aspects, and advantages of the apparatus and 
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methods of the present invention will become better understood with regard to the 
following description, appended claims, and accompanying drawings where: 

Figure 1 illustrates a schematic of a first device for carrying out the 
methods of the present invention. 

Figure 2 illustrates a schematic of a second device for carrying out the 
methods of the present invention. 

Figure 3 illustrates a flow chart of a preferred method according to the 
present invention. 

Although this invention is applicable to numerous and various types of 
display devices, it has been found particularly useful in the environment of televisions. 
Therefore, without limiting the applicability of the invention to televisions, the invention 
will be described in such environment. Those skilled in the art will appreciate that other 
types of display devices which display video and close-captioned text can be utilized in 
the methods and with the devices of the present invention, such as a computer monitor, a 
cellular telephone display, and a personal digital assistant display. 

Referring now to Figure 1, there is illustrated a first device for displaying 
close-captioned text associated with video, the first device being configured as a 
television 100. The television has a display screen 102 such as a CRT, and LCD, or a 
projection screen. The television 100 further has a processor 104 that receives a video 
content (hereinafter referred to simply as "video") input signal 106. The video input 
signal 106 can be from any source known in the art, such as cable, broadcast television, 
satellite, or an external source such as a tuner, VCR, DVD, or set-top box. The processor 
104 is further operatively connected to a storage device 108 for storing data, settings, 
and/or program instructions for carrying out the conventional functions of the television 
100 as well as the methods of the present invention. Although shown as a single storage 
device 108, the same may be implemented in several separate storage devices that may be 
any of many different types of storage devices known in the art. 

The processor 104 receives the video input signal 106, processes the same, 
as necessary, as is known in the art and outputs a signal 1 10 to the display screen in a 
format compatible with the display screen 102. The display screen 102 displays a video 
portion of the video input signal 106. An audio portion 1 12 of the video input signal 106 
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is reproduced on one or more speakers 1 14 also operatively connected to the processor 
104. The one or more speakers 114 may be integral with the television 100, as shown in 
Figure 1 or separable therefrom. The video input signal 106 includes a close-captioned 
text portion for reproducing close-captioned text 1 16 on a portion of the display screen 
102. As is known in the art, a user can program the television 100 through a user 
interface to display the close-captioned text 116. The user may also program the 
language and position of the close-captioned text 1 16 on the display screen 102 with the 
user interface. Absent such programming, the close-captioned text 116 generally defaults 
to a certain language and position, such as English and across a bottom of the display 
screen 102. The use of close-captioned text 1 16 is very useful for people who are hearing 
impaired and in situations where audio is inappropriate, such as locations where the 
television is not the main focus and is viewed in the background, such as in a bar or a 
sports club. 

Referring now to Figure 2, there is shown a second device for displaying 
close-captioned text 116 associated with video, the second device being configured as an 
external source, such as a set-top box, tuner, computer, DVD, or VCR. The external 
source is generally referred to herein by reference numeral 150 and refers generally to 
any device that supplies a video input signal to a display device, such as the television 
100. In the configuration of Figure 2, the television 100 may be as configured in Figure 1 
or it may simply be a monitor under the control of a processor 152 contained in the 
external source 150. Thus, as shown in Figure 2, the input video signal 106 from the 
processor 152 of the external source 150 may be directly input to the display screen 102 
or to the display screen via the television processor 104. The processor 152 is operatively 
connected to a storage device 154 which may be implemented as one or more separable 
storage devices. The storage device 154 includes data and settings as well as program 
instructions for the normal operation of the external source and/or television 100 as well 
as for carrying out the methods of the present invention. 

As will be discussed below, depending upon the configuration of the 
device, the processor 104, 152 determines a position on a portion of the video for display 
of the close-captioned text 116, detects one or more attributes of the video surrounding 
the position, and adjusts one or more attributes of the close-captioned text 116 based on 
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the detected one or more attributes of the video. As discussed above, the position of the 
close-captioned text 116 may be assigned by a default or set by the user, in either way, its 
location can be determined by accessing a location in the storage device 108, 154 where 
such settings are stored. Furthermore, the detection of attributes of video is well known 
in the art, such as determining a color, brightness, contrast, and content of the video by 
analyzing the pixels that make up the video at the desired position. Lastly, the 
adjustment of one or more attributes of the close-captioned text, such as color, brightness, 
contrast, and degree of transparency is also well known in the art, such as assigning the 
pixels which make up the close-captioned text 116 appropriate values, which can be 
taken from appropriate lookup tables, also stored in the storage device 108, 154. After 
the adjustment to the one or more attributes of the close-captioned text 1 16 is made, the 
processor 104, 152 further displays the close-captioned text 1 16 in the portion of the 
video with the adjusted one or more attributes. 

Referring now also to Figure 3, a method for displaying close-captioned 
text associated with video will be described. At step 200 a video input signal is received. 
As discussed above, the video input signal includes close-captioned text corresponding to 
an audio portion of the video. The video signal can be received by any means known in 
the art, such as from cable, television broadcast, satellite, tuner, DVD, or VCR. At step 
202, it is determined whether close-captioning is required either by the user or as a • 
default of the device. If close-captioning is not required, the method proceeds to step 204 
where the video is displayed on the display screen 102 without closed-captioned text. If 
it is determined that close-captioning is required, the method proceeds to step 206 where 
the location of the close-captioned text 116 is determined. Generally, the location of the 
closed-captioned text 116 is predefined and stored in memory, such as in the storage 
device 108, 154. At step 208, one or more attributes of the video surrounding the 
position of the closed-captioned text 1 16 is detected. As discussed above, such attributes 
can be the color, brightness, contrast, and content of the video. The content of the video 
refers to the detection of objects in the video surrounding the position of the close- 
captioned text 116. For example, it may be detected that a person's head is displayed in 
the position surrounding the close-captioned text 116 and that the person's head is the 
most prominent head detected in the video. The detection of the one or more attributes of 
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the video surrounding the close-captioned text 116 can be continuous or sampled at 
predetermined intervals or frames. 

At step 210 it is determined whether one or more of the attributes of the 
close-captioned text 116 needs to be adjusted based on the detected attributes of the video 
surrounding the close-captioned text. If it is determined that the one or more attributes of 
the close-captioned text 116 does not need adjustment, the method proceeds to step 214 
where the video and (unadjusted) close-captioned text are displayed. After step 214, the 
method loops back to step 208 where the video surrounding the close-captioned text 116 
is continually detected and monitored. As discussed above, this determination can be 
made continuously or at certain predetermined intervals or frames. The determination at 
step 208 can also be made only when the close-captioned text 1 16 is about to be replaced 
with new text. Furthermore, the determination at step 208 can include an analysis of 
whether a motion vector from one frame of the video to another frame is above a set 
threshold, thus, signaling an end of one video clip or portion and the start of another, vide 
clip or portion. Techniques for detecting motion and for detecting the beginning and 
ending of video clips are well known in the art. ■.. . ■ 

If it is determined that one or more of the attributes of the close-captioned 
text needs to be adjusted, the method proceeds to step 212, where one or more attributes 
of the close-captioned text 1 16 are adjusted based on the detected attributes of the video 
surrounding the close-captioned text 1 16. The attributes of the close-captioned text are 
generally known to the device, such as being stored in a settings portion of the storage 
device 108, 154. As discussed above, the attributes of the close-captioned text 116, are 
generally set by the device but may be changed by the user through a user interface. 

The determination at step 210 generally involves a comparison of the 
attributes of the close-captioned text 1 16 with the attributes of the video surrounding the 
close-captioned text 116. Any number of ways known in the art can be utilized for 
determining whether an adjustment in the close-captioned text 1 16 is necessary. For 
example, if one or more of the attributes of the close-captioned text 116 differs from a 
corresponding attribute of the video surrounding the close-captioned text by a value less 
than a predetermined threshold. For example, if the color of the close-captioned text has 
a color value very similar to a color value of at least a portion of the color value of the 
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video surrounding the close-captioned text 1 16, the method can determine that an 
adjustment in the color of the closed-captioned text 1 16 is necessary. Similar 
determinations can be made with regard to other attributes such as contrast and 
brightness. Where the attribute of the video surrounding the closed-captioned text is the 
content of the video, the closed-captioned text can be adjusted at step 212 to change its 
degree of transparency to allow the user to view objects through the close-captioned text 
116. In the example described above, the viewer can view the prominent person in the 
video through the transparent closed-captioned text 116. 

The determination at step 210 can be done considering the close-captioned 
text and surrounding video on the whole or in portions thereof. For example, the 
determination can be made for each letter or word in the closed-captioned text 116 and 
the corresponding video surrounding each letter or word. Alternatively, the 
determination at step 210 can be done for the closed-captioned text as a whole, e.g., for 
all the closed captioned text that is to be displayed at any one moment. If the 
determination at step 210 is done for selected portions of the close-captioned text 1 16, . 
any adjustments made to the attributes of the close captioned text 116 should be such that 
a smooth transition is made between adjustments in each of the portions. If the 
determination at step 210 is done on the close-captioned text 1 16 as a whole, any 
adjustment at step 212 made to the attributes of the close-captioned text should be done 
based on all of the video surrounding the closed-captioned text. For example, if the video 
surrounding the close-captioned text contains red, green, and blue pixels, the 
determination to adjust the color of the close-captioned text should not include changing 
the same to either of red, green, or blue. In such a circumstance, the close-captioned text 
116 should be changed to a color different from all of red, green, and blue. Alternatively, 
the change in the color of the close-captioned text 116 can be a similar color that is 
modified by a predetermined factor. For example, if the color of the video surrounding 
the close-captioned text 116 and the color of the close-captioned text are both the same 
color or within a predetermined threshold of the same color (e.g., both are red or very 
similar reds), the color of the close-captioned text 116 can be changed to another red 
within a predetermined factor (e.g., a brick red instead of a cherry red). Similarly, where 
the one or more attributes of the video surrounding the close-captioned text 1 16 is 
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brightness and/or contrast and it is determined at step 210 that the contrast and/or 
brightness of the closed-captioned text 116 needs to be adjusted, the brightness and/or 
contrast of the closed-captioned text can be adjusted by a predetermined factor, such as 
by 50%. For example, if the video surrounding the close-captioned text 116 is very dark 
and the close-captioned text 116 has a high brightness, the brightness of the close- 
captioned text 116 can be reduced by 50% or any other predetermined factor. The 
predetermined factor can be changeable by the user through a suitable user interface. 

It is important to note that when changing any of the attributes of the 
close-captioned text 116, care should be taken such that the perceptive quality of the 
video is not lost. For example, if the color of the video surrounding the position of the 
close-captioned text 1 16 is white, if the color of the close-captioned text 1 16 is changed 
to a dark red, the user could be attracted to the close-captioned text and loose or detract 
from the overall view of the video. Thus, a milder color should be chosen for the close- 
captioned text 1 16 to prevent the user from losing or being distracted from the video. 

After adjustments are made to one or more of the attributes of the close- 
captioned textl 16 at step 212, the method proceeds to step 214 where the close- 
captioned text 116 having the adjusted attributes are displayed at the selected position on 
the video screen 102 along with the corresponding video. The method then loops back to 
step 208 for detection and monitoring of one or more of the attributes of the video 
surrounding the close-captioned text 116. 

The methods of the present invention are particularly suited to be carried 
out by a computer software program, such computer software program preferably 
containing modules corresponding to the individual steps of the methods. Such software 
can of course be embodied in a computer-readable medium, such as an integrated chip or 
a peripheral device. 

While there has been shown and described what is considered to be preferred 
embodiments of the invention, it will, of course, be understood that various modifications 
and changes in form or detail could readily be made without departing from the spirit of 
the invention. It is therefore intended that the invention be not limited to the exact forms 
described and illustrated, but should be constructed to cover all modifications that may 
fall within the scope of the appended claims. 



